Techniques for Evaluating Novice-Oriented Creativity Support Tools

We present evaluation techniques for a novice-oriented creativity support tool in the domain of digital filmmaking. Novices need help executing tasks as well as knowing which tasks are appropriate and the implications their decisions have for their creative product. With our tool, we focus on supporting critical domain knowledge that can enable novices to make meaningful creative contributions. In film, domain knowledge includes cinematographic and editing rules and conventions. Our creativity support tool (CST) provides feedback when novices violate these norms. We use two approaches to evaluate the results: (1) Expert Consensual Assessment, and (2) Individual-Group Consensus, which is a specialized technique we developed to overcome the limitations of the Consensual Assessment Technique for this application. We discuss these two techniques and their domains of application. Finally, we call for further research into evaluation techniques for CSTs sensitive to the task evaluated (e.g. execution vs. knowledge support) and relevant domain (e.g. machinima). Author